DIPS - 11/04/2024
Traditional educational research often fixates on average academic achievement.
Average performance and variability convey distinct information.
How can we identify schools with unusually high or low variability in academic achievement?
We adapt Mixed-Effects Location Scale Model (MELSM) incorporating a spike and slab prior into the scale component to select or shrink random effects.
Based on Bayes factors, we can decide on whether a school is consistent or inconsistent in its academic achievement.
MELSM allows for the simultaneous estimation of a model for the means (location) and a model for the residual variance (scale).
Both sub-models are conceptualized as mixed-effect models.
\[\begin{equation} \textbf{v}= \begin{bmatrix} u_0 \\ t_0 \end{bmatrix} \sim \mathcal{N} \begin{pmatrix} \boldsymbol{0}= \begin{bmatrix} 0 \\ 0 \end{bmatrix}, \boldsymbol{\Sigma}= \begin{bmatrix} \tau^2_{u_0} & \tau_{u_0t_0} \\ \tau_{u_0t_0} & \tau^2_{t_0} \end{bmatrix} \end{pmatrix} \end{equation}\]
Residual variance it’s not merely random noise.
Accounts for possible correlations among location and scale effects.
Allows the inclusion of specific predictors in both sub-models.
We incorporate the Spike-and-Slab prior as a method of variable selection of random effects in the scale model.
The model is allowed to switch between two assumptions:
\[\begin{equation} \color{lightgray}{ \textbf{v}= \begin{bmatrix} u_0 \\ t_0 \end{bmatrix} \sim \mathcal{N}} \begin{pmatrix} \color{lightgray}{ \boldsymbol{0}= \begin{bmatrix} 0 \\ 0 \end{bmatrix},} \boldsymbol{\Sigma}= \begin{bmatrix} \tau^2_{u_0} & \tau_{u_0t_0} \\ \tau_{u_0t_0} & \tau^2_{t_0} \end{bmatrix} \end{pmatrix} \end{equation}\]
\[\begin{equation} \label{eq:cholesky_approach} \textbf{L} = \begin{pmatrix} 1 & 0 \\ \rho_{u_0t_0} & \sqrt{1 - \rho_{u_0t_0}^2} \end{pmatrix} \end{equation}\]
\[\begin{equation} \textbf{v} = \boldsymbol{\tau}\textbf{L}\boldsymbol{z} \end{equation}\]
We include an indicator variable (\(\delta_{jk}\)) for each random effect to be subjected to shrinkage.
It allows switching between the spike and slab throughout the MCMC sampling process.
\[\begin{equation} \begin{aligned} u_{0j} &= \tau_{u_0}z_{ju_0}\\ t_{0j} &= \tau_{t_0}\left( \rho_{u_0t_0}z_{ju_0} + z_{jt_0}\sqrt{1 - \rho_{u_0t_0}^2} \right)\color{red}{\delta_{jt_0}} \end{aligned} \end{equation}\]
Each element in \(\boldsymbol{\delta}_j\) takes integers \(\in \{0,1\}\) and follows a \(\delta_{jk} \sim \text{Bernoulli}(\pi)\) distribution.
When a 0 is sampled, the portion after the fixed effect drops out of the equation.
\[\begin{equation} \label{eq:mm_delta} \sigma_{\varepsilon_{ij}} = \begin{cases} \exp(\eta_0 + 0), & \text{if }\delta_{jt_0} = 0 , \\ \exp(\eta_0 + t_{0j}), & \text{if }\delta_{jt_0} = 1 \end{cases}. \end{equation}\]
If \(\delta= 0\), the density “spikes” at the zero point mass;
If \(\delta= 1\), the standard normal prior, \(z_{jk}\), is retained and scaled by \(\tau_k\), introducing the “slab”.
\[\begin{align} \label{eq:pip_theorical} Pr(\delta_{jk} = 1 | \textbf{Y}) = \frac{Pr(\textbf{Y} | \delta_{jk} = 1)Pr(\delta_{jk} = 1)}{Pr(\textbf{Y})} \end{align}\]
\[\begin{align} \label{eq:pip} Pr(\delta_{jk} = 1 | \textbf{Y}) = \frac{1}{S} \sum_{s = 1}^S \delta_{jks} \end{align}\]
If there is evidence for zero variance in the scale random effects, the model reduces to the MLM assumption:
\[\varepsilon_{ij}\sim\mathcal{N}(0, \sigma_\varepsilon)\]
If not, the MELSM assumption of variance heterogeneity is retained:
\[\varepsilon_{ij}\sim\mathcal{N}(0, \sigma_\varepsilon_{ij})\]
\[\begin{align} \label{eq:bf_pip} BF_{10j} = \frac{Pr(\delta_{jk} = 1 | \textbf{Y}) }{1 - Pr(\delta_{jk} = 1 | \textbf{Y}) } \end{align}\]
We use a subset of data from the 2021 Brazilian Evaluation System of Elementary Education (Saeb).
It focuses on math scores from 11th and 12th-grade students across 160 randomly selected schools, encompassing a total of 11,386 students.
The analysis compares three SS-MELSM models with varying levels of complexity:
The model was fitted using ivd package in R (Rast & Carmo, 2024).
All models were fitted with six chains of 3,000 iterations and 12,000 warm-up samples.
We computed the estimation efficiency using \(\hat{R}\) and the effective sample size (ESS).
The models were compared for predictive accuracy using Pareto smoothed importance sampling Leave-one-out cross-validation (PSIS-LOO).
Model 1 identified eight schools with PIPs exceeding 0.75, suggesting notable deviations from the average within-school variance.
By incorporating SES covariates, Model 2 significantly outperformed Model 1 in terms of predictive accuracy, \(\Delta\widehat{\text{elpd}}_{\text{loo}}= -43.6 (10.5)\) .
Model 3 was practically indistinguishable from Model 2; the inclusion of a random slope for the student-level SES did not improve the model’s predictive accuracy, \(\Delta\widehat{\text{elpd}}_{\text{loo}}= -1.3 (0.6)\).
The SS-MELSM helps identifying schools deviating from the norm in terms of within-school variability.
The spike-and-slab prior accounts for uncertainty in including random effects.
Identifying variability can guide resource allocation or teaching interventions.
Currently, the SS-MELSM demands significant computational resources, especially with bigger datasets or more complex models.
It is still not clear how model performance is affected by the choice of hyperparameters.
Further development could explore the method’s performance in longitudinal data settings.
Beyond Averages with MELSM and Spike-and-Slab